Internal server error loading a page with a header containing non-ASCII characters

[fred-wang]

./wpt serve and open http://localhost:8000/common/blank.html?pipe=header(Content-Security-Policy,trusted-types%20%E3%83%9D%E3%83%AA%E3%82%B7%E3%83%BC)

WARNING - Internal server error loading http://localhost:8000/common/blank.html?pipe=header(Content-Security-Policy,trusted-types%20%E3%83%9D%E3%83%AA%E3%82%B7%E3%83%BC):
  Traceback (most recent call last):
    File "web-platform-tests/tools/wptserve/wptserve/server.py", line 301, in finish_handling
    handler(request, response)

    File "web-platform-tests/tools/wptserve/wptserve/handlers.py", line 228, in __call__
    response = wrap_pipeline(path, request, response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/handlers.py", line 164, in wrap_pipeline
    response = Pipeline(pipe_string)(request, response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/pipes.py", line 42, in __call__
    response = func(request, response, *args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/pipes.py", line 135, in inner
    return f(request, response, *arg_values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/pipes.py", line 188, in header
    response.headers.set(name, value)

    File "web-platform-tests/tools/wptserve/wptserve/response.py", line 363, in set
    value = _maybe_encode(value)
            ^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/response.py", line 347, in _maybe_encode
    return isomorphic_encode(s)
           ^^^^^^^^^^^^^^^^^^^^

    File "web-platform-tests/tools/wptserve/wptserve/utils.py", line 40, in isomorphic_encode
    return s.encode("iso-8859-1")
           ^^^^^^^^^^^^^^^^^^^^^^
  UnicodeEncodeError: 'latin-1' codec can't encode characters in position 14-17: ordinal not in range(256)

[fred-wang]

cc @jgraham

[annevk]

It seems like a problem that it first does a UTF-8 decode (presumably) and then a latin1 encode. If it just took the percent-encoded bytes as given we wouldn't run into this.

I would recommend working around this by using a custom Python script to serve the response instead and not wait for pipes to be fixed.